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Abstract 


In  this  report,  we  discuss  an  assessment  methodobgy  developed  to  assist  in  determining  whether 
we  are  winning  or  losing  the  global  war  ofterror  in  Operation  Enduring  Freedom.  The 
assessment  system  developed  is  based  on  metrics  and  was  developed  with  the  support  of  the 
Operational  level  staff  in  Afghanistan. 

From  the  start  of  the  operation,  the  Combined  Joint  Task  Force  -  180  in  Afghanistan  needed  a 
methodology  to  assess  their  current  situation.  The  system  had  to  provide  them  with  a  means  to 
convey  their  status  to  higher  levels  of  command  and  support  decisions  on  future  necessary 
effects  required  by  subordinates.  In  late  2002,  the  command  asked  a  group  from  the  Information 
Technology  Operations  Center  (ITOC)  from  the  US  Military  Academy  to  develop  a  system  to 
assist  in  this  process. 

At  that  time,  the  command  wanted  a  system  that  allowed  a  great  deal  of  subjective  assessments 
and  only  loosely  based  on  quantitative  analysis.  To  support  this,  the  ITOC  developed  a  system 
called  the  Dynamic  Planning  and  Assessment  System,  or  DPASS.  This  web-based  system 
decomposed  the  effects  the  command  was  asked  to  attain  by  the  CENTCOM  CONOP.  The 
system  was  very  useful  for  the  command  at  the  time. 

In  spring  of  2003,  Mr.  Donald  Rumsfeld  announced  that  Afghanistan  had  entered  Phase  IV,  or 
Stability  Operations.  This  new  mission  set  coincided  with  the  arrival  of  a  new  command  group 
in  CJTF-180.  They  immediately  sought  a  system  based  directly  on  metrics.  The  command  staff 
could  not  adjust  DPASS  to  directly  account  for  these  and  they  decided  to  ask  the  Operations 
Research  Center  of  Excellence  (ORCEN)  also  at  the  US  Military  Academy  to  look  at  the 
problem  for  them. 

The  ORCEN  analysts,  in  consultation  with  the  previous  ITOC  analysts  and  the  CJTF-180 
Assessment  Staff,  determined  that  a  major  obstacle  in  the  development  of  a  quantitative  means  to 
assess  the  implementation  of  the  effects-based  orders  was  the  “task-based”  approach  to 
decomposition.  Our  functional  decomposition,  the  identification  of  direct  metrics  and  the 
application  of  value  focused  thinking  to  the  assessment  process  are  discussed  in  this  presentation 
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and  paper.  Additionally,  the  model  developed  and  now  being  implemented  in  CJTF-180  will  be 
discussed  in  detail.  The  model,  the  Afghanistan  National  Development  Assessment  System,  or 
ANDAS,  achieves  Secretary  Rumsfeld’s  goal  of  being  able  to  use  metrics  to  assess  our  level  of 
success  as  well  as  provides  the  CJTF-180  staff  with  a  means  to  identify  on  which  effects  they 
should  focus  their  efforts  in  future  operations. 
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Chapter  1:  Introduction 


Today,  we  lack  metrics  to  know  if  we  are  winning  or  losing  the  global 
war  on  terror.  Are  we  capturing,  killing  or  deterring  and  dissuading  more 
terrorists  every  day  than  the  madrassas  and  the  radical  clerics  are  recruiting, 
training  and  deploying  against  us? 

—  The  Honorable  Donald  Rumsfeld 

(“Rumsfeld's  war-on-terror  memo”,  USA  Today,  Oct  22,  2003) 

Timely  and  accurate  assessment  is  a  significart  part  of  any  military  operation.  Proper 
assessments  provide  commanders,  at  all  levels,  with  necessary  information  to  rapidly  adjust 
resources  and  missions  to  meet  ever-changing  battlefield  requirements.  Unfortunately,  for  many 
reasons,  developing  assessments  in  a  battlefield  environment  is  a  difficult  and  complex  task. 

One  complicating  factor  is  the  inherent  inaccuracies  and  time  delays  in  battlefield  reporting. 
Another,  albeit  associated,  factor  that  makes  assessment  such  a  complex  task  is  the  inherent 
difficulty  that  staff  have  in  determining  exactly  what  information  is  required  to  provide 
“actionable”  assessments.  Actionable  assessments  allow  commanders  to  adjust  their  resources 
and  missions  in  a  manner  which  best  supports  the  achievement  of  current  objectives  and  results 
in  the  desired  effects. 

The  above  quotation  from  the  Secretary  of  Defense  underscores  the  current  push  in  the 
US  Military  to  develop  quantitative  measures  to  support  proper  assessments.  Despite  this 
growing  awareness  of  the  importance  of  quantitative  assessments  of  battlefield  operations,  there 
is  some  reluctance  to  move  away  from  qualitative  assessments  in  current  operations.  This 
reluctance  is  borne,  at  least  in  part,  of  the  reliance  of  the  US  Government  and  the  media  on  what 
are  now  know  to  be  exaggerated  body  counts  during  in  the  Vietnam  conflictto  assess  the  success 
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of  the  conflict  there.  Based  on  this  measure  and  others  which  were  historically  used  to  assess 
previous  conflicts,  the  United  States  clearly  “won”  that  war.  It  is  clear  however  that  these 
measures  did  not  provide  an  accurate  portrayal  of  this  conflict. 

The  key,  therefore,  in  providing  useful  and  actionable  quantitative  assessments  to 
commanders  leading  forces  involved  in  Operations  Enduring  Freedom  in  Afghanistan  and  Iraqi 
Freedom  in  Iraq  is  determining  the  correct  measures  to  accurately  assess  progress  in  those 
regions.  In  this  paper,  we  discuss  two  ongoing  efforts  that  were  directed  by  the  commanders  and 
the  staffs  of  the  headquarters  operating  in  Afghanistan  to  develop  an  assessment  system  designed 
to  direct  future  operations  and  to  report  their  level  of  success  to  their  higher  headquarters,  the  US 
Central  Command  (CENTCOM). 

The  first  effort,  conducted  very  early  in  the  conflict,  was  led  by  the  Information 
Technology  and  Operations  Center  or  Excellence  (ITOC)  from  the  United  States  Military 
Academy  at  West  Point.  The  Combined  Joint  Task  Force  -  180  (CJTF-180)  has  been  the  on-the- 
ground  command  since  early  in  the  conflict  in  Afghanistan.  Very  early  on,  the  command  group 
recognized  the  need  for  a  comprehensive  assessment  system  for  this  operation.  Analysts  from 
the  ITOC  were  asked  by  the  command  to  travel  to  Afghanistan  to  develop  a  distributed,  web- 
based  system  to  perform  this  critical  function.  The  system  they  developed  was  dubbed  the 
Dynamic  Planning  and  Assessment  Support  System,  or  D-PASS.  At  the  request  of  the  group 
then  in  command,  this  system  relied  on  a  more  qualitative  means  to  assess  the  operation.  D- 
PASS  served  the  command  extremely  well  for  the  early  phases  of  the  operation. 

The  assumption  of  the  command  by  a  different  unit  and  the  progress  ofthe  operation  into 
a  different  phase  of  the  operation  resulted  in  a  desire  to  make  significant  changes  to  the 
assessment  process.  The  most  significant  change  requested  by  the  new  command  group  was  a 
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greater  reliance  on  quantitative  assessments.  This  change  of  focus  resulted  in  the  command 
requesting  the  participation  of  the  Operations  Research  Center  of  Excellence  (ORCEN).  The 
ORCEN,  like  the  ITOC,  is  one  of  many  Centers  of  Excellence  (COE)  at  the  United  States 
Military  Academy,  each  of  which  has  different  focuses  and  competencies. 

This  follow-on  effort  combined  the  competencies  of  the  ORCEN  and  the  ITOC  to 
develop  an  assessment  system  which  combined  the  quantitative  assessment  desired  by  the  new 
command  with  the  distributed,  web-based  features  of  the  initial  D-PASS  system.  This  new 
system  was  dubbed  the  Afghanistan  National  Development  Assessment  System,  or  ANDAS. 

In  this  paper,  we  begin  in  the  next  section  with  a  description  of  the  first  effort  by  the 
ITOC  in  its  development  of  D-PASS  and  its  application  in  theater.  In  the  following  section,  we 
describe  the  process  followed  by  the  ORCEN,  with  support  of  a  senior  analyst  from  the  Air 
Force  Institute  of  Technology  (AFIT),  in  the  development  of  the  quantitative  assessment  of  the 
operation  and  the  subsequent  collaboration  with  the  ITOC  in  the  eventual  development  of 
ANDAS.  The  following  section  describes  in  detail  the  prototype  system  developed  for  the 
operational  command.  We  conclude  with  a  summary  of  the  current  status  of  the  system  and 
discuss  potential  applications  of  the  process  and/or  the  product. 

Chapter  2:  The  Dynamic  Planning  and  Assessment 
Support  System  (D-PASS) 

In  early  June  2002,  the  Director  of  the  Staff  of  the  Combined  and  Joint  Task  Force-180, 
stationed  in  Bagram,  Afghanistan  made  a  request  to  the  US  Military  Academy  (USMA)  for  five 
officers  to  fill  critical  positions  on  the  CJTF  Planning  staff,  CJ-5.  The  Academy  responded 
quickly  to  this  request  and  within  a  week  an  officer  from  each  of  five  different  academic 
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departments  was  identified  and  had  begun  the  deployment  process.  The  specific  mission  given 
to  these  officers  was  not  specified  until  the  group  arrived  in  Afghanistan  in  mid  June.  That 
mission,  the  development  of  a  comprehensive  and  distributed  process  for  performing  operational 
assessments,  became  the  focus  not  only  for  these  five  officers  but  for  ten  faculty  members  from 
the  USMA  Department  of  Electrical  Engineering  and  Computer  Science.  This  squad  of 
additional  faculty  members,  both  military  and  civilian,  made  significant  contributions  to  the 
design  and  implementation  of  the  fully  distributed  web-based  application  that  came  to  be  known 
as  D-PASS. 

The  CJTF  staff,  and  in  particular,  Lieutenant  Colonel  James  Dickens,  the  Deputy 
Director  of  Plans,  CJTF- 180,  specified  a  number  of  important  requirements  for  the  system  to  the 
D-PASS  development  team.  These  were  requirements  that  they  did  not  feel  were  met  by  their 
existing  non-automated  process.  In  addition,  they  did  not  feel  that  their  requirements  were  met 
by  other  existing  software  systems,  including  the  CENTCOM-developed  Campaign  Analysis  / 
Decision  Support  System  (CA/DSS)  (Carlock,  2002).  For  the  record,  it  should  be  noted  that 
while  the  CA/DSS  did  not  meet  the  needs  of  the  CJTF-1 80  staff  many  of  the  underlying  ideas  in 
D-PASS,  particularly  the  employment  of  a  hierarchical  arrangement  of  effects,  objectives,  and 
tasks,  were  inspired  by  the  CA/DSS  application. 

The  stated  requirement  for  D-PASS  was  that  it  “provide  an  automated  means  to  develop 
and  refine  effects-based  strategic,  operational,  and  tactical  assessments  to  support  the  planning 
process.”  (Dickens,  2003)”  In  more  detail,  the  CJTF-1 80  staff  requested  that  the  D-PASS  team 
design  and  implement  an  application  that  supported  an  assessment  process  for  the  CJTF  staff  that 
was  rigorous,  consistent,  and  well-coordinated.  Significant  effort  was  expended  on  the  part  of 
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D-PASS  developers  in  Afghanistan  and  at  West  Point  to  ensure  that  the  resulting  application  met 
these  requirements. 

Two  early  design  decisions  proved  to  be  critical  to  the  success  of  the  project.  These 
decisions  included  using  an  uncomplicated  web  interface  and  incorporating  the  use  of  a 
relational  database.  Because  the  application  uses  a  relational  database,  the  many  important 
relationships  between  various  areas  of  assessmert  could  be  readily  represented  and  captured. 

This  database  “back-end”  also  allowed  staff  officers  to  provide  greater  justification  for  their 
assessments  for  other  users  of  the  system;  particularly  the  Commander.  The  distributed  nature  of 
the  application  was  fully  facilitated  through  the  use  of  a  simple  web-interface.  This  allowed 
CJTF  staff  officers  who  were  responsible  for  assessment  to  directly  collaborate  with  their 
counterparts  in  other  staff  sections  on  the  CJTF  staff  as  they  performed  their  respective 
assessment.  If  should  be  noted  that  prior  to  the  development  of  D-PASS,  operation  assessments 
performed  by  CJTF  staff  officers  were  done  using  ad-hoc  methods  that  provided  only  limited 
justification  for  assessment  to  the  CJTF-180  Commander  and  higher  headquarters. 

Another  benefit  of  D-PASS  was  that,  in  addition  to  serving  as  an  analysis  tool,  its 
uncomplicated,  graphical  web  interface  allowed  it  to  serve  as  an  excellent  presentation  tool.  For 
nearly  eighteen  months  the  CJTF  Commander  received  weekly  or  bi-weekly  briefings  in  which 
D-PASS  was  the  primary  presentation  medium.  This  meant  that  very  little  time  was  devoted  by 
staff  officers  translating  information  from  D-PASS  to  a  presentation  graphics  application  (i.e., 
Microsoft  PowerPoint). 

The  D-PASS  team  was  able  to  successfully  address  many  challenges  during  the 
development  process  including:  the  need  to  develop  and  test  an  unclassified  application  that 
processes  primarily  classified  data,  and  the  12-hour  time  difference  between  Afghanistan  and 
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New  York.  One  challenge  that  proved  intractable  to  the  team  was  the  inability  of  the  team  to 
incorporate  quantitative  analysis  into  the  assessment  process.  There  are  clearly  areas  where  such 
analysis  would  contribute  to  a  better  overall  assessment  but,  unfortunately,  the  D-PASS  team 
was  unable  to  develop  consensus  among  the  users  of  the  system  concerning  the  manner  in  which 
this  analysis  should  be  done.  Consequently,  the  final  prototype  for  D-PASS,  while  considered  an 
unqualified  success  by  all  of  it  initial  users,  provided  almost  no  support  for  quantitative  analysis. 

Chapter  3:  The  Afghanistan  National  Development 
Assessment  System  (AND AS) 

By  mid-July  2003  it  had  become  clear  to  the  newly  arrived  command  group  for  CJTF- 
180  that  D-PASS  was  not  providing  them  with  the  assessment  means  they  desired.  They  began 
to  press  their  assessment  team,  part  of  the  CJ5  staff  section,  to  providethem  with  quantitative 
metrics  to  support  their  assessments  of  the  operation.  Mounting  frustrations  by  the  command 
group,  as  well  as  the  assessment  group,  prompted  the  Chief,  Fires  and  Effects  staff  section  to 
offer  the  services  of  an  organization  with  which  he  had  worked  in  his  previous  assignment  to 
assist  in  the  development  of  a  more  quantitative  assessment  tool  for  that  command. 

The  Chief,  Fires  and  Effects  section  at  the  time  was  Colonel  Cieighton,  the  Commander 
of  the  Division  Artillery  for  the  10th  Mountain  Division.  Prior  to  this  command  assignment, 
Colonel  Creighton  was  the  Chief  of  G8,  the  analytical  support  section  of  the  United  States  Army 
Staff.  While  in  tljis  assignment,  he  worked  on  a  project  to  determine  the  best  alternative  to  field 
the  Army’s  newest  weapon  system,  the  Future  Combat  System  (FCS)  which  was  still  under 
development.  Recognizing  the  need  to  provide  a  quantitative  underpinning  for  his 
recommendation  to  the  Army  Chief  of  Staff,  he  contacted  the  ORCEN  at  the  United  States 
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Military  Academy.  Being  very  impressed  with  their  support  of  thatanalysis,  he  quickly 
recommended  their  participation  on  developing  a  quantitative  assessment  system  to  the  CJTF- 
180  command  group.  By  early  August,  a  small  two-man  team  from  the  ORCEN,  augmented  by 
a  senior  analyst  from  AFIT,  departed  to  Afghanistan  to  develop  the  quantitative  assessment 
system  desired  by  the  command. 

3. 1  Interim  Assessment  Process 

In  the  summer  of  2003,  the  responsibility  for  developing  assessment  reports  for  CJTF- 
180  fell  squarely  on  the  shoulders  of  the  CJTF-180  Assessment  Team  Chief,  Major  Gus  Kostas. 
Major  Kostas,  a  Marine  Reserve  officer,  had  been  working  assessments  for  CJTF-180  since  his 
arrival  in  country  the  previous  November.  His  presentations  to  the  command  had  become 
increasing  difficult.  The  combination  of  the  pressure  from  his  superiors  for  quantitative  evidence 
for  the  assessments  and  the  lack  of  support  from  the  other  staff  sections  in  developing  the  weekly 
assessments  had  clearly  begun  to  wear  on  him. 

To  help  develop  the  assessment  report  for  the  command,  Major  Kostas  had  organized  an 
assessment  team.  The  team  consisted  of  representaives  from  the  Civil-Military  Operations  Staff 
Section  (CJCMOTF),  the  Intelligence  Section  (CJ2),  the  Fires/Effects  Section  (CJTF  Fires),  and 
the  Information  Operations  Section  (CJTFIO).  The  assessment  process  consisted  of  convening 
the  group  and  discussing  in  detail  one  sub-task  from  D-PASS.  There  was  a  great  deal  of 
discussion  and  subjective  assessment  included  in  the  overall  assessment.  This  part  of  the  process 
normally  took  two  hours  and  required  meetings  two  or  three  times  a  week. 

These  assessment  meetings  would  be  the  first  step  in  the  overall  assessment  process. 

After  updating  the  task/objective,  Major  Kostas  would  take  the  partial  assessment  and  roll  it  into 
the  overall  assessment  that  he  had  completed  for  the  remaining  tasks.  His  requests  for 
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information  from  the  other  sections  usually  were  not  answered.  This  putgreat  pressure  on  Major 
Kostas  to  develop  a  reasonable,  and  defensible,  assessment  of  the  Afghanistan  operation.  Once 
prepared,  the  assessment  would  then  be  presented  to  the  Command  Group  by  Major  Kostas. 

3.2  Initial  Meetings  and  Directions 

When  the  team  first  met  with  Major  Kostas,  he  was  unconvinced  that  the  team  could 
develop  a  system  the  command  group  would  like  and  the  staff  could  use.  Of  primary  concern  for 
Major  Kostas  was  that  the  command  group  did  not  clearly  express  their  priorities  between  the 
often  conflicting  directions  of  the  assessment  systems.  This  created  an  atmosphere  of  everything 
being  important  and  therefore  limited  the  ability  of  the  assessment  team  to  provide  useful 
information  for  the  Effects  team  in  the  CJTF-180  planning  cell. 

There  was  also  a  significant  initial  discussion  on  exactly  what  the  Director  of  the  Staff  of 
CJTF-180  wanted  the  team  to  accomplish.  There  were  some  on  the  staff  who  wanted  to  the  team 
to  concentrate  on  developing  assessment  for  Information  Operations.  There  were  others  who 
wanted  the  team  to  work  on  assessments  for  only  the  combat  portions  of  the  conflict  in 
Afghanistan.  The  team  quickly  decided  that  they  needed  to  start  at  the  top  to  quickly  focus  on 
the  “right”  problem. 

They  met  with  BG  Byron  Bagby,  the  Director  of  the  Staff  for  CJTF-180  within  two  days 
of  their  arrival.  BG  Bagby  provided  the  team  with  the  focus  they  needed  to  begin  the  process  of 
developing  an  assessment  tool.  He  said  that  the  command  needed  to  be  able  to  explain  how  they 
were  doing  and  base  it  on  quantitative  evidence.  They  also  needed  to  be  able  to  determine  on 
what  they  should  focus  their  efforts  in  the  future  of  the  conflict. 

The  team  explained  they  intended  to  conduct  stakeholder  interviews  with  all  command 
section  chiefs,  unit  commanders  and  with  individuals  involved  in  the  military-political  efforts  in 
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Kabul.  They  would  also  develop  a  system  for  meeting  the  Directors  requirements,  would  depart 
in  three  weeks  and  then  return  to  Afghanistan  with  the  finished  product.  The  Director  supported 
the  process  going  forward  and  the  timeline. 

3.3  Stakeholder  Analysis 

The  first  step  the  team  undertook  after  meeting  with  the  Director  was  to  interview  each 
significant  stakeholder  in  the  assessment  process  for  the  command  to  ensure  they  captured  all  the 
concerns  in  the  system  they  developed.  This  also  provided  a  direct  way  to  ensure  each 
stakeholder  that  the  system  was  not  being  developed  to  make  their  jobs  more  difficult,  or  of  more 
concern  to  the  staff  section  chiefs,  provide  a  means  for  the  Director  to  directly  determine  how 
their  sections  were  performing. 

The  initial  intent  was  to  interview  each  staff  section  chief,  each  commander  including  the 
commanders  in  the  field,  and  the  military-political  assets  in  Kabul.  Major  Kostas,  who 
responsibility  it  was  to  make  arrangements  for  all  the  interviews,  was  reluctant  to  leave  Bagram 
Airbase  for  outside  interviews.  His  rationale  was  that  others  could  do  the  interfacing  with  the 
stakeholders  outside  the  base  and  that  there  was  not  enough  time  for  the  interviews.  The  team 
reluctantly  agreed.  This  turned  out  to  be  a  poor  decision. 

The  list  below  summarizes  the  concerns  of  the  stakeholders  interviewed.  Essentially,  the 
system  should  be  able  to: 

-  provide  a  means  to  measure  success  along  three  lines  of  operations  over  time. 

-  be  measurable. 

-  be  simple  to  provide  inputs. 

-  address  both  national  and  regional  impacts. 

-  provide  a  means  to  differentiate  between  CJTF,  ISAF  and  GOA/UNAMA/ 
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NGOs/IOs/USG  and  others’  efforts. 


-  allow  subjective  assessments  (not  just  management  by  statistics). 

-  provide  traceability  from  existing  orders  and  other  documents  and  be  tractable  to  inputs 
from  organizations  within  the  Afghan  AOR 

-  reflect  items  that  are  critical  to  Afghan  culture  and  not  just  to  Western  ideals. 

-  consider  coalition  access  and  input. 

3.4  Value  Approach  to  Effects  Based  Operations  Analysis 

When  the  second  assessment  team  arrived  in  early  August,  the  Phase  IV  order  had  been 
already  published  by  the  US  Army  Central  Command  (CENTCOM).  Phase  IV  is  generally 
referred  to  as  “Stability  Operations”.  This  phase  is  normally  considered  to  begin  only  after  all 
combat  operations  have  ended.  In  a  non-traditional  operational  environment  like  Afghanistan, 
combat  operations  had  not  ended  when  the  US  Army  Secretary  of  Defense  announced  the 
operation  had  transitioned  to  Phase  IV.  In  spite  of  this  announcement,  the  new  order  included 
both  a  combat  Line  of  Operation  as  well  as  Lines  of  Operations  focused  on  the  stability  of  the 
new  Afghani  government  and  gaining  the  support  of  the  populace. 

Corps-level  operations,  which  the  CJTF  was  considered,  are  widely  termed,  “the 
Operational  Level”  of  combat  operations.1  At  this  level,  higher  headquarters  orders  directed  the 
subordinate  headquarters  to  “achieve  effects”  vice  specific  objectives  like  terrain  features.  These 
“Effects-Based  Operations”  were  the  cause  of  most  of  the  assessment  problems.  Assessing 
success  in  controlling  terrain  features  was  simple  compared  to  achieving  stability  in  a  region. 
Since  the  D-PASS  structure  was  based  on  a  task-criented  breakdown  structure,  it  was  not  well- 


'Below  Corps-level  is  considered  the  “Tactical  level”  and  above  the  Corps  level  is  considered  the  “Strategic  level”. 
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suited  to  assess  the  achievement  of  these  effects  quantitatively  because  the  structure  did  not 
accommodate  summation  from  lower  levels  of  the  hierarchy. 

To  counter  this,  the  team  began  developing  a  functional  decomposition.  This  effort 
would  attempt  to  functionally  decompose  a  desired  effect  completely.  Then  each  function  would 
be  decomposed  into  sub-functions.  This  would  continue  until  the  sub-functions  could  be  directly 
measured  using  attainable  Measures  of  Effectiveness  (MOEs).  Because  the  objective  of  the 
assessment  system  was  to  assess  the  attainment  of  the  overall  effect,  the  measures  had  to  be 
summarized.  This  created  a  problem  in  that  the  units  of  the  individual  measures  were  different 
which  prevented  direct  summing  of  the  sub-measures.  To  address  this,  the  team  turned  to  “value 
modeling”  (Keeney,  1992). 

Value  modeling  is  generally  used  to  analyze  between  competing  alternatives  and  is  a 
favorite  approach  in  decision  analysis.  In  this  approach,  the  analyst  develops,  in  conjunction 
with  a  client,  a  “value  curve”.  This  curve  is  intended  to  be  the  functional  transformation  of  the 
measure  of  effectiveness  to  a  value  between  0  and  10.  The  shape  of  these  curves  is  important 
and  must  be  decided  on  based  on  the  relative  level  of  importance  of  increasing  the  level  of  the 
measure.  The  four  basic  examples  of  the  shapes  of  the  curves  are  shown  in  figure  1  below: 


Figure  1.  Four  types  of  value  functions  (Parnell,  2003) 
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After  the  development  of  the  hierarchy  including  the  measures  and  establishment  of  the 
value  functions,  the  decision  maker  identifies  weights  for  each  function,  sub-function  and 
eventually  each  value.  These  values,  which  sum  to  one  for  each  level,  is  then  multiplied  by  the 
value  which  leads  to  a  “score”  at  each  level.  In  the  application  of  thisapproach  to  assessment 
systems,  the  weights  represent  the  commander’s  priorities  for  the  desired  effects  from  the 
CENTCOM  order.  The  first  step  and  most  difficult  step  would  be  to  develop  the  hierarchy.  This 
was  done  as  a  group  effort. 

3. 5  Mission  Deconstruction  and  Assessment  System 

Development 

The  assessment  development  team  and  Major  Kostas  gathered  a  group  of  experts  from 
the  entire  staff  to  help  develop  the  functional  decomposition  hierarchy  of  the  effects.  There  were 
representatives  from  the  Intelligence  section,  the  Information  Operations  section,  the 
Fires/Effects  section,  the  Plans  section,  the  Development  aid  Reconstruction  Interagency 
Council  (DARIC)  section,  the  Civil-Military  Operations  section  and  others. 

The  process  they  followed  was  simple.  For  each  effect  the  operations  order  directed 
(called  Lines  of  Operation),  the  group  would  identify  functions  that  must  be  achieved  in  order  to 
successfully  attain  the  overall  effect.  The  functions  that  the  group  established  where  then 
grouped  together.  These  groups  become  the  next  lower  level  of  the  hierarchy.  This  process 
continues  until  the  sub-function  can  be  directly  measured  by  one  or  a  few  measures  of 
effectiveness.  This  is  a  time-consuming  and  painstaking  process.  The  benefit  is  that  as  a  group, 
the  functions  are  more  comprehensive  and  there  is  more  “buy-in”  as  the  process  continues.  The 
ability  of  the  facilitator  is  critical  to  keep  the  group  on  task. 
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After  two  weeks  of  work,  the  hierarchy  was  about  80%  complete.  At  this  point,  the  team 
returned  to  West  Point  for  completion  of  the  hierarchy  and  the  development  of  the  overall 
assessment  system.  Unfortunately,  the  analysts  departed  the  area  without  interviewing  any 
members  from  the  embassy  in  Kabul  or  any  members  of  the  Office  of  Military  Cooperation  - 
Afghanistan  (OMC-A)  which  is  responsible  to  support  the  bourgeoning  Afghani  government. 

3. 6  Interim  Model  Development 

Upon  the  assessment  team’s  return  to  West  Point  in  early  September,  the  Information 
Operations  and  Technology  Center  (ITOC)  and  the  Operations  Research  Center  (ORCEN)  begin 
working  directly  together  on  the  model  for  the  first  time.  The  plan  to  complete  the  task  was  a 
two-phase  operation.  In  the  first  phase,  the  model  hierarchy  would  be  completed  and  integrated 
into  an  Excel-based  model.  This  interim  product  would  be  then  returned  to  Afghanistan  for 
evaluation.  In  the  second  phase,  the  evaluated  model  would  be  developed  into  a  web-based 
system  which  would  be  fully  distributed  with  the  functionality  of  DPASS. 

The  intent  of  the  system  was  to  sum  the  weighted  values  derived  from  the  transformation 
of  the  measures  of  effectiveness  for  each  level.  The  weighted  values  would  then  be  calculated 
for  each  level  of  the  value  hierarchy.  The  scores  themselves  for  a  given  month  would  be  of  little 
value  in  assessing  the  progress  of  the  campaign.  What  would  be  of  interest  would  be  the 
comparison  of  a  monthly  value  to  the  values  of  proceeding  months.  In  this  way,  trend  analysis 
would  provide  insight  into  the  direction  of  the  effects.  It  would  also  highlight  those  areas  which 
would  provide  the  most  benefit  in  subsequent  months.  These  would  be  the  effects  on  which  the 
command  should  focus  their  efforts  in  future  operations. 

The  hierarchy  was  completed  and  the  Excel-based  model  was  completed  by  the  end  of 
September  and  sent  to  Afghanistan  so  the  assessment  section  could  begin  collecting  data.  The 
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assessment  team  from  West  Point  would  follow  in  early  October  to  determine  required  changes 
and  to  present  the  model  formally  to  the  Director  of  the  Staff. 

Upon  their  arrival,  they  were  made  aware  of  changes  which  occurred  in  the  command 
while  they  were  away.  First,  another  level  of  command  was  established  in  Afghanistan.  CJTF- 
1 80  now  fell  under  the  control  of  the  Combined  Forces  Command  -  Afghanistan  (CFC-A), 
eventually  commanded  by  MG(P)  Barno.  CJTF-180  was  now  commanded  by  BG(P)  Austin, 
while  BG  Bagby  remained  as  the  Director  of  the  CJTF  -180  staff.  In  addition,  Major  Kostas 
would  soon  be  replaced  by  Major  McCullough,  another  Reserve  Marine. 

The  impact  of  the  new  command  structure  would  soon  be  felt.  When  the  Assessment 
team  from  West  Point  briefed  the  new  system  to  BG  Bagby  and  BG(P)  Austin  it  was  well 
received  as  a  comprehensive  means  of  assessing  the  progress  of  the  command  using  quantitative 
means  —  exactly  the  stated  mission  of  the  team.  The  presentation  to  the  new  commander 
however  did  not  go  quite  as  well. 

MG(P)  Barno  did  not  like  the  process  nor  the  product  presented  to  him  by  the  team.  His 
primary  point  of  contention  with  the  process  was  that  the  team  only  interviewed  military 
personnel  and  did  not  interact  with  any  political  personnel  from  the  Embassy.  This  was  a  valid 
point  and  a  significant  weakness  of  the  study  (identified  earlier)  especially  since  one  of  the 
primary  effects  the  command  hoped  to  achieve  was  purely  political  in  nature. 

Other  issues  that  MG(P)  Bamo  disagreed  with  were  the  basics  of  the  system 
development.  He  did  not  feel  that  summing  the  values  for  each  effect  provided  the  correct  view 
of  the  actual  situation.  Also,  he  did  not  want  to  become  so  tied  to  quantitative  measures  but 
rather  wanted  to  use  qualitative,  subjective  assessments  from  subordinate  commanders  to  a 
greater  extent. 
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The  team  returned  to  BG  Bagby  to  report  on  their  less-than-successfiil  reception  by 
MG(P)  Barno.  BG  Bagby,  who  was  the  client  for  the  team’s  work,  wanted  the  team  to  continue 
to  the  final  development  of  the  model  into  a  distributed,  web-based  assessment  system.  He  felt 
that  though  the  system  in  its  current  formulation  was  not  deemed  useful  at  the  CFC-A  level,  it 
was  still  of  use  at  the  CJTF-180  level.  With  this,  the  team  returned  to  West  Point  to  complete  the 
final  version  of  the  system. 

3. 7  Final  Model  Development 

The  team  initially  felt  that  there  was  sufficient  functionality  incorporated  in  the  original 
DPASS  programming  to  integrate  the  weighted  sum  and  trend  analysis  of  the  new  system.  As 
development  progressed  however,  it  soon  became  clear  that  changing  the  original  code  would  be 
as  problematic  as  a  complete  construction  of  a  new  system. 

This  new  system  is  a  fully  distributed,  web-based  replication  of  thefimctionality  of  the 
Excel-based  system.  It  offers  trend  analysis,  the  capability  to  provide  subjective  assessments  by 
subordinate  sections  and  commanders  at  all  levels  and  a  means  to  change  the  functions,  sub- 
functions  and  measures  without  reprogramming  the  entire  system.  At  figure  2  below  is  a  screen 
shot  of  the  developmental  opening  page  of  the  system. 
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Figure  2.  Opening  Page  of  the  Web-based  ANDAS 
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The  ease  of  viewing  of  the  status  of  the  trend  of  the  sub-functions  is  seen  at  figure  3, 
below.  In  this  screen  shot,  one  sees  a  notional  Line  of  Operation  and  the  trend  for  this  LOO. 


Note  the  inclusion  of  the  subjective  assessments  below  the  LOO  description  and  the 
decomposition  of  the  LOO  on  the  right  side  of  the  screen. 
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Figure  3.  Notional  Line  of  Operation  Assessment 


The  measures  under  the  higher  level  functions  and  LOOs  are  listed  on  the  right  hand  side 
of  the  screen.  The  user  can  enter  at  any  level  and  see  the  trend  of  the  function,  the  LOO  or  the 
measure  over  time.  This  will  allow  the  analyst  to  determine  which  effects  they  should 
recommend  the  command  focus  their  efforts  in  future  operations. 

The  measures  will  be  input  by  each  responsible  staff  section  on  their  own  web  page.  The 
sections  will  sign  into  the  system  and  can  view  the  status  of  all  the  functions,  but  can  only 
modify  the  value  of  the  measures  associated  with  their  sections.  Only  the  administrator,  usually 
the  Assessment  Team  Chief,  will  be  allowed  to  modify  the  weights,  the  functions  or  sub- 
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functions,  and  the  actual  measures.  This  will  help  prevent  sections  from  modifying  the  weights 
to  change  the  trend  results. 

Chapter  4:  Future  Direction  of  Research 

The  system  is  ready  to  be  fully  deployed  in  its  web-based  configuration,  though  there 
remains  some  reluctance  on  the  part  of  the  command  in  Afghanistan  to  integrate  this  into  their 
assessment  system.  The  system  is  nearly  fully  verified  in  its  functionality.  There  must  be  a  time 
for  validation  of  the  system  prior  to  being  able  to  rely  on  it  to  provide  sufficient  insights  to  direct 
future  effects.  We  anticipate  this  taking  only  one  or  two  months.  The  reason  is  that  the 
command  has  been  compiling  most  of  the  data  required  and  will  be  able  to  generate  data  for  past 
months  allowing  good  insights  into  the  system. 

The  system,  as  it  is,  is  deemed  to  have  too  many  measures  to  be  useful  long  term.  It  may 
take  too  long  for  staff  sections  to  input  the  data  to  ensure  that  the  system  is  fully  maintained. 

The  LOOs  within  the  Area  of  Operation  in  Afghanistan  have  already  changed  and  will  continue 
to  change.  There  needs  to  be  a  means  to  allow  for  quick  changes  to  the  system.  There  will  never 
be  a  “best”  means  to  assess  the  situation  for  two  reasons.  First,  tie  situation  changes  and 
therefore  the  system  must  change  to  provide  good  insights  to  direct  effects.  Second,  this  system 
is  designed  to  provide  insights  to  allow  the  commander  to  make  decisions,  therefore  as  the 
commander  changes  and  the  information  needed  to  make  those  decisions,  the  system  will  have  to 
change. 
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Chapter  5:  Conclusion 


Developing  one  comprehensive  assessment  system  to  analyze  an  entire  theater  of 
operations  was  a  challenging  proposition  for  the  analysts.  In  the  end,  changing  the  command 
structure  changed  the  complexion  of  the  problem  and  resulted  in  the  limited  acceptance  of  the 
product.  This  is  a  strong  system  based  directly  on  quantitative  analysis  to  provide  the 
commander  with  the  means  to  make  more  informed  decisions.  This  is  the  role  of  the  military 
analyst  in  supporting  a  combat  commander. 

The  military  analyst’s  role  is  not  always  easy.  As  this  study  shows,  there  are  some  who 
are  very  wary  of  quantitative  assessments  and  methods.  These  are  not  always  trusted  because 
there  is  a  belief  they  can  be  manipulated  and  that  they  can  actually  hide  reality.  The  analyst  must 
convince  the  commander  that  any  system  is  reliable  and  at  the  same  time  is  only  there  to  provide 
the  commander  with  the  means  to  make  more  informed  decisions. 

The  reality  is  that  we  live  in  a  political  time  and  one  where  the  media  has  unprecedented 
access  to  information  within  a  military  command.  Any  system  which  provides  definitive  results 
-  whether  they  accurately  reflect  the  situation  or  are  just  open  for  interpretation  -  can  possibly  be 
used  against  the  military  or  used  politically.  Commanders  must  be  sensitive  to  this  reality. 
Analysts  must  also  be  sensitive  to  this  reality  but  we  must  provide  the  commander  with  what  we 
believe  is  the  best  product  possible.  The  next  decision  lies  withthe  commander,  but  the  final 
decision  will  be  made  by  history. 
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